On the Accuracy of Stochastic

Stochastic complexity of a data set is deened as the shortest possible code length for the data obtainable by using some xed set of models. This measure is of great theoretical and practical importance as a tool for tasks such as determining model complexity, or performing predictive inference. Unfortunately for cases where the data has missing information, computing the stochastic complexity requires margin-alizing (integrating) over the missing data, which results even in the discrete data case to computing a sum with an exponential number of terms. Therefore in most cases the stochastic complexity measure has to be approximated. In this paper we will investigate empirically the performance of some of the most common stochastic complexity approximations in an attempt to understand their small sample behavior in the incomplete data framework. In earlier empirical evaluations the problem of not knowing the actual stochastic complexity for incomplete data was circumvented either by using synthetic data, or by comparing the behavior of the stochastic complexity approximation methods to crossvalidated prediction error, approaches which both suuer from validity problems. Our comparison is based on the novel idea of using demonstrably representative small samples from real data sets, and then calculating by \brute force" the exponential sums. This allows for the rst time a comparison between the true stochastic complexity and its approximations with real-world data.